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Why  GAO  Did  This  Study  What  GAO  Found 


Hurricane  Katrina  caused  several 
breaches  in  the  floodwalls  along 
three  drainage  canals  in  New 
Orleans,  contributing  to 
catastrophic  flooding.  To  restore 
the  pre-Katrina  level  of  hurricane- 
related  flood  protection,  the  Army 
Corps  of  Engineers  (Corps) 
decided  to  acquire  several  large- 
capacity  pumping  systems.  During 
the  process  of  acquiring,  testing, 
and  installing  the  pumping  systems, 
issues  with  the  pump  contract  and 
operation  of  the  pumping  systems 
came  to  light,  including  several 
identified  in  a  Corps  Independent 
Team  Report  (ITR). 

GAO  was  asked  to  evaluate  the 
Corps’  efforts  to  (1)  develop 
contract  specifications  and  award 
the  contract,  (2)  address  pumping 
system  performance  issues,  (3) 
document  contract  modifications, 
and  (4)  reconcile  contract 
payments.  GAO  reviewed  contract 
and  testing  documents,  observed 
the  operation  of  the  pumping 
system,  and  interviewed  officials 
from  the  Corps,  its  consultants  and 
contractors,  and  the  ITR  team. 


What  GAO  Recommends 


GAO  recommends  actions  to 
ensure  that  future  contracts  adhere 
to  sound  acquisition  practices, 
even  for  expedited  procurements, 
and  to  ensure  that  any  required 
contract  documentation  is 
completed  and  filed  in  a  timely 
manner. 

The  Department  of  Defense  agreed 
with  GAO’s  recommendations. 


To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-08-288. 

For  more  information,  contact  Anu  Mittal  at 
(202)  512-3841  or  mittala@gao.gov. 


Schedule  concerns  drove  the  Corps’  decisions  in  developing  specifications  for 
the  pumping  systems  and  awarding  the  contract,  but  the  rush  to  award  the 
contract  resulted  in  deficiencies  in  key  contract  provisions.  Specifically,  the 
original  factory  test  requirements  were  ambiguous,  there  were  only  limited 
provisions  for  on-site  testing,  and  there  were  no  criteria  for  acceptance  of  the 
pumping  systems  by  the  government.  The  Corps  conducted  an  expedited 
competition  to  contract  for  the  pumping  systems  and  selected  a  supplier  for 
contract  award  based  largely  on  its  ability  to  deliver  the  pumping  systems  by 
the  June  I  start  of  the  2006  Atlantic  hurricane  season. 

The  Corps  and  the  contractors  have  addressed  and  corrected  known 
performance  issues  with  the  pumping  systems.  Concerns  included  hydraulic 
motor  vibrations,  the  design  of  the  hydraulic  intake  line,  suspect  pipe  welds, 
and  lower  than  expected  pumping  capacity.  The  pumping  systems  were 
installed  prior  to  correcting  these  issues  because  the  Corps  believed  it  was 
better  to  have  some  pumping  capacity  along  the  drainage  canals  during  the 
2006  hurricane  season  rather  than  none,  despite  uncertainty  over  how  much 
of  the  pumping  system  capacity  would  be  available,  and  for  how  long,  if 
needed.  Between  November  2006  and  September  2007,  the  Corps  and  the 
contractors  completed  all  of  the  repairs  and  reinstalled  the  pmnping  systems. 
Documents  that  GAO  reviewed  indicate  that,  as  of  September  2007,  each 
piunping  system  had  been  successfully  tested  on  site  for  at  least  2  homs,  thus 
providing  greater  assurance  that  they  will  perform  as  designed. 

The  contract  files  for  the  pumping  systems  contained  the  required 
docmnentation  for  the  type  and  value  of  the  contract  and  associated 
modifications,  though,  in  a  number  of  cases,  dociunentation  was  inserted  in 
the  contract  files  several  months  after  modifications  were  issued  and  only 
after  the  ITR  reported  its  findings.  While  the  ITR  correctly  noted  the  absence 
of  some  required  documentation,  GAO  found  that  much  of  the  specific 
docmnentation  cited  as  missing  was  not  required  for  the  modifications  in 
question  because  of  the  nature  and  value  of  these  modifications.  In  addition, 
while  the  ITR  foimd  that  it  appeared  as  though  the  contractor  developed  the 
scope  of  work  and  pricing  for  some  of  the  modifications  without  a  subsequent 
analysis  by  the  Corps,  GAO  found  no  instance  of  this  occurring. 

As  of  October  31,  2007,  the  Corps  had  paid  the  contractor  about  $30.5  million 
of  the  $33  million  contract  amount.  In  a  few  instances,  the  Corps  made 
duplicate  payments  to  the  contractor.  GAO  foimd  that  these  payments  were 
due  to  Corps  mistakes,  not  inappropriate  billing  by  the  contractor.  GAO  found 
no  other  cases  of  duplicate  payments.  The  Corps  plans  to  adjust  for  the 
duplicate  payments  by  deducting  the  balance  from  remaining  funds,  including 
any  incentive  payments,  owed  to  the  contractor.  According  to  Corps  officials, 
final  payment  and  reconciliation  of  the  contract  is  expected  by  early  2008; 
however,  it  is  unknown  to  what  extent  contract  or  pump  performance  issues 
will  affect  the  final  amount  paid  for  the  contract  dming  the  close-out  process. 
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Accountability  *  Integrity  *  Reliability 


United  States  Government  Acconntability  Office 
Washington,  DC  20548 


December  31,  2007 

The  Honorable  Mary  L.  Landrieu 
Chairman 

Ad  Hoc  Subcommittee  on  Disaster  Recovery 

Committee  on  Homeland  Secmity  and  Governmental  Affairs 

United  States  Senate 

Dear  Madam  Chairman: 

The  greater  New  Orleans  metropolitan  area  sits  in  the  tidal  lowlands  of 
Lake  Pontchartrain  and  is  bordered  generally  on  its  southern  side  by  the 
Mississippi  River  and  the  Gulf  of  Mexico.  Due  to  the  city’s  location  and 
elevation,  which  averages  six  feet  below  sea  level,  the  area  has  historically 
experienced  many  floods.  In  an  effort  to  reduce  such  floods,  a  series  of 
levees  and  other  flood  control  structures  have  been  built  over  the  years  to 
reduce  that  threat.  However,  hurricane-induced  storm  surges,  waves,  and 
rainfall  continue  to  pose  a  threat  to  New  Orleans.  To  avoid  flooding  in 
New  Orleans  from  a  rain  storm,  the  city’s  Sewerage  and  Water  Board 
pumps  rainwater  from  the  city  into  three  drainage  canals  located  at  17th 
Street,  London  Avenue,  and  Orleans  Avenue,  which  then  flows 
unrestricted  into  Lake  Pontchartrain.  While  critical  to  prevent  flooding 
from  rainfall,  these  canals  are  vulnerable  to  storm  smge  from  Lake 
Pontchartrain  during  hmricanes;  consequently,  floodwalls  have  been 
erected  along  both  sides  of  the  canals  to  protect  against  storm  surge 
overtopping  the  canals  and  flooding  the  city. 

On  August  29,  2006,  storm  surge  from  Hurricane  Katrina  caused  several 
breaches  in  the  floodwalls  along  the  17th  Street  and  London  Avenue 
canals,  contributing  to  catastrophic  flooding  in  New  Orleans.  In  its  efforts 
to  restore  pre-Katrina  levels  of  hmricane  protection  to  New  Orleans  by  the 
Jime  1  start  of  the  2006  Atlantic  hurricane  season  (which  generally  runs 
from  June  1  to  November  30  each  year),  the  U.S.  Army  Corps  of  Engineers 
(Corps),  in  late  2005,  considered  strengthening  the  drainage  canal 
floodwalls  but  decided  against  this  effort  due  to  cost  and  time  constraints. 
Instead,  the  Corps  decided  to  install  three  interim  closure  structures  or 
gates  near  the  points  where  the  canals  meet  the  lake.  These  gates  would 
be  closed  dming  major  hmricane  events  to  prevent  storm  surge  from 
entering  the  canals  and  potentially  overtopping  or  breaching  the  canal 
floodwalls  and  flooding  the  city.  When  the  gates  are  closed,  however, 
rainwater  cannot  drain  from  the  three  canals  into  Lake  Pontchartrain,  and 
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large-capacity  pumping  systems  are  needed  to  pump  water  out  of  the 
canals  and  into  the  lake.  Due  to  space  constraints  along  the  canals  and  the 
limited  amount  of  time  it  had  before  the  start  of  the  2006  hurricane  season, 
the  Corps  decided  to  procure  34  large-capacity  hydraulically  powered 
pumping  systems^  to  provide  the  most  pumping  capacity  possible  by  June 
1,  2006.  In  mid-2006,  the  Corps  modified  the  contract  to  procure  six 
additional  hydraulic  pumping  systems,  bringing  the  total  number  of 
hydraulic  pumping  systems  to  be  installed  along  the  three  drainage  canals 
to  40." 

During  the  process  of  acquiring,  testing,  and  installing  the  hydraulic 
pumping  systems  for  the  drainage  canals,  issues  with  the  operation  of 
these  pumping  systems  came  to  light.  In  response  to  your  request,  we 
issued  a  report  on  May  23,  2007,  on  the  procurement  process  and  award  of 
the  pumping  system  contract  and  the  status  of  the  efforts  to  address  issues 
related  to  the  performance  of  the  pumping  systems."  In  June  2007,  the 
Corps’  Mississippi  Valley  Division  (MVD)  released  a  technical  review 
report,  known  as  the  Independent  Team  Report  (ITR).  This  report 
identified  testing  and  performance-related  issues  regarding  the  hydraulic 
pumping  systems  similar  to  the  issues  we  had  identified  in  our  May  report 
and,  in  addition,  raised  potential  issues  related  to  the  administration  and 
documentation  of  contract  modifications.  The  ITR  was  conducted  by  a 
three-person  technical  review  team,  which  consisted  of  two  Corps 
engineers  from  outside  MVD  and  one  engineer  from  MVD  who  acted  as  the 
team  coordinator.  According  to  ITR  members,  the  New  Orleans  District 
was  not  offered  the  opportunity  to  provide  official  comments  on  the  draft 
ITR;  rather,  an  advisory  panel  from  MVD  reviewed  the  draft  before  its 
issuance  in  June  2007. 

In  this  context,  you  asked  us  to  update  our  May  2007  report  and  consider 
the  ITR  findings  in  our  analysis.  Specifically,  you  asked  us  to  evaluate  the 


^Unlike  a  more  typical  direct  drive  water  pumping  system  where  the  motor  or  engine  is 
directly  coupled  by  a  shaft  to  the  pump  it  is  turning,  a  hydrauhc  pumping  system  is  one 
where  pressurized  hydraulic  oil  is  used  to  transmit  power  from  an  engine  to  a  pump 
impeller.  This  allows  for  greater  flexibility  in  the  placement  and  possible  isolation  of  the 
engine  from  the  pumping  system  because  they  cire  connected  by  hydraulic  lines. 

"During  the  2007  hurricane  season,  the  Corps  also  installed  14  portable  hydrauhc  pumps 
along  the  17th  Street  Canal  and  19  additional  direct  drive  pumps  from  another  pump 
manufacturer  along  the  17th  Street  and  London  Avenue  Canals. 

"GAO,  U.S.  Army  Corps  of  Engineers’  Procurement  of  Pumping  Systems  for  the  New 
Orleans  Drainage  Canals,  GAO-07-908R  (Washington,  D.C.:  May  23,  2007). 
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Corps’  efforts  to  (1)  develop  the  specifications  for  the  pumping  systems, 

(2)  award  the  contract,  (3)  address  pumping  system  performance  issues 
identified  during  factory  and  on-site  testing,  (4)  document  modifications  to 
the  contract,  and  (5)  reconcile  payments  made  and  amounts  still  owed  to 
the  contractor. 

To  address  these  objectives,  we  reviewed  the  Corps’  plans  for  the  interim 
gates  and  temporary  pumping  systems  for  the  three  New  Orleans  drainage 
canals.  (Unless  otherwise  noted,  the  pumping  systems  discussed  in  this 
report  are  only  the  40  60-inch  hydraulic  pumps  installed  at  the  three 
canals.)  We  reviewed  documentation,  including  e-mails,  correspondence, 
and  other  documents  related  to  the  solicitation  process,  contract 
specifications  and  other  provisions,  factory  and  on-site  test  results, 
performance  requirements,  contract  modification  files,  payments  to  the 
contractor,  and  the  Corps’  plans  for  increasing  pumping  capacity  through 
2007.  We  visited  the  17th  Street,  London  Avenue,  and  Orleans  Avenue 
Canals  and  observed  the  operation  of  the  pumping  systems.  We 
interviewed  officials  from  (1)  Corps  Headquarters,  New  Orleans  District, 
other  Corps  districts,  and  members  of  the  MVD’s  technical  review  team 
related  to  the  contract  and  pump  performance;  (2)  Moving  Water 
Industries  (MWI)  Corporation  and  two  other  pump  suppliers  that  bid  on 
the  solicitation;  and  (3)  the  architectural  and  engineering  consulting  firms 
under  contract  with  the  Corps  that  researched  available  pumping  system 
alternatives,  qualified  pump  manufacturers,  pump  delivery  timelines,  and 
costs  and  that  helped  design  the  canal  gates  and  pumping  stations.  We 
conducted  our  work  from  September  through  December  2007  in 
accordance  with  generally  accepted  government  auditing  standards. 


Results  in  Brief 


Schedule  concerns  drove  the  Corps’  decisions  in  developing  specifications 
for  the  pumping  systems,  but  the  rush  to  award  the  contract  resulted  in 
deficiencies  in  key  contract  provisions.  The  Corps  was  committed  to 
having  as  much  pumping  capacity  as  possible  in  place  at  the  drainage 
canals  by  June  1,  2006 — the  start  of  the  Atlantic  hurricane  season.  Due  to 
the  compressed  schedule  and  the  limited  space  available  for  installation, 
and  based  on  the  limited  market  research  conducted  by  the  Corps’ 
consultants,  the  Corps  decided  to  use  60-inch  hydraulic  pumping  systems 
rather  than  alternatives  that  would  have  involved  longer  delivery 
schedules  or  required  more  space.  The  Corps’  consultants  drafted  contract 
specifications  that  closely  matched  those  of  one  supplier,  which, 
combined  with  the  60-inch  pumping  system  requirement,  resulted  in  that 
supplier  being  in  the  strongest  position  to  compete  for  the  contract. 
Further,  the  contract  itself  was  not  written  as  precisely  as  it  should  have 
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been.  Specifically,  the  original  factory  test  requirements  were  ambiguous, 
there  were  limited  provisions  for  on-site  testing,  and  there  were  no  criteria 
for  acceptance  of  the  pumping  systems  by  the  government. 

Given  the  need  to  procure  and  install  the  temporary  pumping  systems 
before  the  start  of  the  hurricane  season,  the  Corps  conducted  an 
expedited  competition  to  contract  for  the  pumping  systems.  The  Corps 
issued  a  competitive  solicitation  and  received  three  proposals.  Using 
established  evaluation  factors,  the  Corps  rated  MWI’s  proposal 
significantly  higher  than  the  other  two  proposals.  The  Corps  selected  MWI 
for  contract  award,  in  large  part,  because  it  determined  that  MWI  was  the 
competitor  most  likely  to  enable  the  agency  to  meet  its  June  1  deadline. 

The  Corps  and  the  contractors  have  addressed  and  corrected  known 
performance  issues  with  the  pumping  systems.  As  identified  in  our  May 
2007  report  and  in  the  ITR,  both  factory  and  on-site  testing  revealed 
several  concerns  with  some  components  of  the  pumping  systems.  Specific 
concerns  identified  during  testing  included  undersized  gear  oil  circulation 
motors,  hydraulic  motor  vibrations,  the  design  of  the  hydraulic  intake  line, 
suspect  pipe  welds,  and  lower  than  expected  pumping  capacity. 
Nevertheless,  the  pumping  systems  were  installed  as  planned  because  the 
Corps  believed  it  was  better  to  have  some  pumping  capacity  along  the 
drainage  canals  during  the  2006  hurricane  season  rather  than  none.  The 
Corps  also  believed  that  many  of  the  issues  identified  during  factory 
testing  could  be  resolved  after  installation.  Consequently,  on  June  1,  2006, 
the  Corps  had  installed  11  pumping  systems,  and  by  July  2006,  it  had 
installed  34.  However,  both  our  May  2007  report  and  the  ITR  concluded 
that  it  was  uncertain  how  much  of  the  pumping  systems’  theoretical 
capacity  would  be  available,  and  for  how  long,  if  needed  during  the  2006 
hurricane  season.  The  Corps  and  its  contractors  have  since  taken  several 
steps  to  correct  known  performance  issues  with  the  pumping  systems. 
These  issues  were  addressed  by  replacing  components  that  were 
undersized,  such  as  motors  and  springs;  redesigning  and  replacing  the 
hydraulic  intake  lines;  rewelding  critical  structural  welds;  and  conducting 
additional  pumping  capacity  tests.  Between  November  2006  and 
September  2007,  the  Corps  and  its  contractors  completed  all  of  the  repairs 
that  were  noted  in  the  ITR  and  reinstalled  all  40  pumping  systems.  As  of 
September  2007,  each  pumping  system  has  been  successfully  tested  on  site 
for  at  least  2  hours,  thus  providing  greater  assurance  that  they  will 
perform  as  designed.  According  to  Corps  officials,  all  of  the  outstanding 
repairs  have  been  completed  and  on-site  testing  indicates  that  the  system 
is  now  fully  operational. 
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Contract  files  for  the  pumping  systems,  although  incomplete  at  the  time  of 
the  ITR  review,  now  contain  the  required  documentation  for  the  type  of 
contract  and  value  of  the  associated  modifications.  In  a  number  of  cases. 
Corps  officials  inserted  required  documentation  in  the  contract  files 
several  months  after  modifications  were  issued  and  only  after  the  ITR 
reported  its  findings.  The  ITR  correctly  noted  the  absence  of  some 
required  documentation.  However,  we  found  much  of  the  documentation 
that  the  ITR  specifically  cited  as  missing — including  requests  for 
proposals,  independent  government  estimates,  certified  cost  or  pricing 
data,  technical  analyses,  and  price  negotiation  memorandums — was  not 
required,  either  because  documentation  was  not  relevant  to  the  contract 
modifications  in  question,  or  the  value  of  the  modifications  were  below 
specified  regulatory  thresholds.  In  addition,  while  the  ITR  found  that  it 
appeared  as  though  the  contractor  developed  the  scope  of  work  and 
pricing  for  some  of  the  modifications  without  a  subsequent  analysis  by  the 
Corps,  we  found  no  instance  of  this  occurring.  Rather,  our  review  found 
that,  for  most  of  the  contract  modifications,  there  was  evidence  of  some 
analysis  by  the  Corps  and  extensive  back  and  forth  discussion,  usually  by 
e-mail,  between  officials  from  the  Corps  and  MWI. 

As  of  October  31,  2007,  the  Corps  had  paid  the  contractor  about  $30.5 
million  of  the  $33  million  contract  for  the  40  hydraulic  pumping  systems 
and  has  since  planned  to  reconcile  mistaken  payments  it  made.  The  Corps 
made  payments  to  the  contractor  after  receiving  invoices  for  items 
delivered,  such  as  drive  units,  pumps,  and  services.  In  most  instances,  the 
Corps  retained  about  20  percent  of  each  invoice  to  ensure  that  the 
contractor  was  not  overpaid.  The  ITR  identified  a  few  instances  where  the 
Corps  made  duplicate  payments  to  the  contractor.  Our  review  found  that 
these  duplicate  payments  involved  mistakes  by  the  Corps,  not 
inappropriate  billing  actions  on  the  part  of  the  contractor.  We  found  no 
additional  cases  where  the  Corps  made  duplicate  payments  to  the 
contractor.  According  to  the  Corps  contracting  officer,  the  duplicate 
payments  will  be  corrected  by  deducting  the  balance  from  retained  funds 
or  by  not  paying  outstanding  invoices.  In  addition,  the  contract  also  has  an 
incentive  clause  of  up  to  $5  million  for  early  delivery,  but  the  Corps  has 
withheld  any  payment  until  the  final  acceptance  of  the  pumping  systems 
occurs.  The  contract  also  has  a  penalty  for  late  delivery.  The  Corps  will 
determine  whether  or  not  a  penalty  will  be  assessed  as  part  of  the  close¬ 
out  process.  According  to  Corps  officials,  final  payment  and  reconciliation 
of  the  contract,  including  any  incentive  payments,  will  be  completed  after 
final  acceptance  of  the  pumping  systems. 
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While  most  of  the  issues  identified  to  date  related  to  testing,  pump  system 
performance,  and  payments  have  been  addressed  by  the  parties,  there  may 
still  be  issues  that  arise  and  need  to  be  resolved  during  contract  closeout. 
The  Corps  expects  contract  closeout  to  take  place  during  the  early  part  of 
calendar  year  2008. 

We  are  recommending  that  the  Corps  develop  procedures  to  help  ensure 
that  all  future  contracts,  including  those  awarded  for  expedited 
procurements,  contain  the  terms  and  conditions  needed  to  ensure  that 
items  contracted  for  meet  the  government’s  needs,  and  that  key  contract 
actions  are  adequately  documented  in  a  timely  manner.  In  commenting  on 
a  draft  of  this  report,  the  Department  of  Defense  concurred  with  our 
recommendations. 


Background 


Since  its  founding  in  1718,  the  city  of  New  Orleans  and  its  surroimding 
areas  have  been  subject  to  numerous  floods  from  the  Mississippi  River  and 
hurricanes.  The  greater  New  Orleans  metropolitan  area,  composed  of 
Orleans,  Jefferson,  St.  Charles,  St.  Bernard,  and  St.  Tammany  parishes,  sits 
in  the  tidal  lowlands  of  Lake  Pontchartrain  and  is  bordered  generally  on  its 
southern  side  by  the  Mississippi  River  and  the  Gulf  of  Mexico.  Lake 
Pontchartrain  is  a  tidal  basin  about  640  square  miles  in  area  that  connects 
with  the  Gulf  of  Mexico  through  Lake  Borgne  and  the  Mississippi  Sound. 

Many  hurricanes  have  struck  the  area  over  the  years,  including  Hurricane 
Betsy  in  1966,  Hurricane  Camille  in  1969,  Hurricane  Lili  in  2002,  and 
Hurricane  Katrina  in  2005.  The  hurricane  surge  that  can  inundate  coastal 
lowlands  is  the  most  destructive  characteristic  of  hurricanes  and  accounts 
for  most  of  the  lives  lost  from  hurricanes.  Because  of  such  threats,  a  series 
of  flood  control  structures,  including  concrete  floodwalls  and  levees,  have 
been  constructed  in  and  around  the  New  Orleans  metropolitan  area  (see 
fig.  1). 
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Figure  1 :  Flood  Protection  Control  Levees  in  and  around  New  Orleans,  Louisiana 


Barriers  of  Earth  and  Concrete 

Levees  and  floodwalls  that  protect  against  flooding  from  both  the  Mississippi  River 
and  hurricanes  are  built  by  the  U.S.  Army  Corps  of  Engineers  and  are  maintained  by 
local  levee  districts.  The  Corps  and  the  local  districts  share  the  construction  cost  of 
hurricane  levees,  while  the  Mississippi  River  levees  are  a  federal  project.  Local  levee 
districts  also  build  and  maintain  nonfederal,  lower-elevation  levees  with  construction 
money  from  each  district's  share  of  property  taxes  and  state  financing. 
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Source:  Staff  graphic  by  Emmett  Mayer  I l!/emayer@ timespicayune.com. 


On  August  29,  2005,  Hurricane  Katrina  came  ashore  near  Buras,  Louisiana, 
about  60  miles  southeast  of  New  Orleans,  with  wind  speeds  of  up  to  127 
miles  per  hour  and  a  storm  driven  wave  surge  of  up  to  30  feet.  The  size  and 
strength  of  the  storm  and  subsequent  flooding  resulted  in  one  of  the 
largest  natural  disasters  in  U.S.  history.  Storm  waters  overtopped 
floodwalls  and  levees  in  Louisiana’s  Orleans  and  neighboring  parishes, 
causing  widespread  flooding,  many  billions  of  dollars  of  property  damage. 
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and  more  than  1,300  deaths.  The  Corps  estimates  that  more  than  one-half 
of  the  269  miles  of  federally  constructed  levees  and  floodwalls  in  these 
parishes  were  damaged  by  the  storm’s  winds  and  floodwaters. 

Through  a  combination  of  permanent  and  temporary  measures,  the  Corps 
planned  to  restore  the  level  of  hurricane  protection  to  the  New  Orleans 
area  that  existed  prior  to  Hurricane  Katrina  by  June  1,  2006.  To  restore  the 
pre-Katrina  level  of  protection  in  a  period  of  about  9  months,  the  Corps 
had  to  work  quickly  and,  in  some  instances,  engineer  temporary  solutions 
because  not  all  of  the  repairs  could  be  completed  in  time.  One  such 
temporary  solution  was  needed  along  the  Orleans  East  Bank,  located 
south  of  Lake  Pontchartrain,  from  the  17th  Street  Canal  to  the  Inner 
Harbor  Navigation  Canal,  and  along  the  western  bank  of  the  Irmer  Harbor 
Navigation  Canal  to  the  Mississippi  River.  About  19  miles  of  levees  and 
floodwalls  are  located  along  the  Orleans  Lakefront,  the  Inner  Harbor 
Navigation  Canal,  and  three  drainage  canals — 17th  Street,  London  Avenue, 
and  Orleans  Avenue — ^which  drain  rainwater  from  New  Orleans  into  Lake 
Pontchartrain.  A  total  of  about  1  mile  of  levees  and  floodwalls  was 
damaged  along  the  17th  Street  Canal  and  two  sides  of  the  London  Avenue 
Canal,  resulting  in  flooding  of  New  Orleans  (see  fig.  2). 
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Figure  2:  Orleans  East  Bank  in  New  Orleans,  Louisiana,  and  Repair  Project  Sites 


Code 

Project 

OEB01 

17th  St.  Canal  Floodwall  Breach 
•  Phase  1  -  Interim  repairs 

OEB02 

17th  St.  Canal  Floodwall  Breach 
•  Phase  II  -  Inverted  T-wall  construction 

0EB03 

London  Ave.  Canal  Floodwall  Breach, 

M  irabeau  Ave. 

•  Phase  1  -  Interim  repairs 

OEB04 

London  Ave.  Canal  Floodwall  Breach, 

M  irabeau  Ave. 

•  Phase  II  -  Inverted  T-wall  construction 

OEB05 

London  Ave.  Canal  Floodwall  Breach, 
RobertE.  Lee  Blvd. 

•  Phase  1  -  Interim  repairs 

OEB06 

London  Ave.  Canal  Floodwall  Breach, 
RobertE.  Lee  Blvd. 

•  Phase  II  -  Inverted  T-wall  construction 

OEB07 

Lake  Pontchartrain  and 

Orleans  Pump  Station 
•  Scour  repair  and  slope  paving 

OEB09 

Interim  Closure  Structure 
at  17th  St.  Canal 

•  Construction  and  installation  of  interim 

OEB10 


gates  and  pumps 
Interim  Closure  Structure 
at  London  Ave.  Canal 

•  Construction  and  installation  of  interim 
gates  and  pumps 

Interim  Closure  Structure 
at  Orleans  Ave.  Canal 

•  Construction  and  installation  of  interim 
gates  and  pumps 

Orleans  Ave.  Canal  Levee  Tie-In  to 
Closure  Structure 

•  Levee  lift,  seepage,  and  scour  protection 

London  Ave.  Canal  Levee  Tie-In  to 
Closure  Structure 

•  Levee  lift,  seepage,  and  scour  protection 


i 


Source:  U.S.  Army  Corps  of  Engineers. 


The  city’s  three  drainage  canals  are  critical  to  avoid  flooding  in  New 
Orleans  from  a  rain  storm.  During  rain  events,  the  city’s  Sewerage  and 
Water  Board  pumps  rainwater  from  the  city  into  three  drainage  canals  at 
17th  Street,  London  Avenue,  and  Orleans  Avenue,  which  then  flows 
imrestricted  into  Lake  Pontchartrain.  According  to  the  Corps,  the 
maximum  amormt  of  water  that  the  Sewerage  and  Water  Board  can  pump 
into  these  drainage  canals  is  10,500  cubic  feet  per  second  (cfs)  at  the  17th 
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Street  Canal,  7,980  cfs  at  the  London  Avenue  Canal,  and  2,690  cfs  at  the 
Orleans  Avenue  Canal.  Because  permanent  structures  and  repairs  could 
not  be  completed  on  the  three  drainage  canals  by  Jrme  1,  2006,  the  Corps 
decided  to  install  temporary  pumping  systems  to  provide  protection  to  the 
area  for  3  to  5  years  until  permanent  structures  can  be  constructed  (see 
fig.  3).  The  Corps  chose  to  install  three  gates  and  temporary  pumping 
systems  near  the  points  where  the  17th  Street,  London  Avenue,  and 
Orleans  Avenue  drainage  canals  meet  Lake  Pontchartrain.  These  gates  are 
intended  to  stop  hurricane-induced  storm  surge  from  Lake  Pontchartrain 
from  entering  the  canals  and  possibly  overtopping  or  breaching  the  canal 
floodwalls,  which  would  flood  the  city.  However,  because  the  gates 
prevent  the  drainage  canals  from  draining  water  from  the  city  into  the  lake 
when  the  gates  are  closed  during  a  hurricane  event,  temporary  pumping 
systems  are  needed  to  pump  water  out  of  the  canals  and  into  the  lake. 


Figure  3:  Graphic  of  Interim  Gates  and  Temporary  Pumps 


Source:  U.S.  Army  Corps  of  Engineers. 
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Due  to  the  hurricane  damage  sustained  by  the  floodwalls  bordering  the 
canals,  the  Corps  established  the  following  safe  water  levels  for  each  of 
the  drainage  canals — 6  feet  for  the  17th  Street  Canal,  5  feet  for  the  London 
Avenue  Canal,  and  8  feet  for  the  Orleans  Avenue  Canal.  The  water  level  in 
each  of  these  canals  must  be  maintained  at  or  below  the  safe  water  level  in 
order  to  ensure  that  the  already  weakened  canal  floodwalls  are  not 
breached.  Further,  the  total  capacity  of  the  temporary  pumping  systems  at 
the  interim  gated  closure  structures  that  is  necessary  to  accommodate  a 
10-year  rainfall  evenC  without  exceeding  the  safe  water  levels  is  7,700  cfs 
at  the  17th  Street  Canal,  5,000  cfs  at  the  London  Avenue  Canal,  and  1,900 
cfs  at  the  Orleans  Avenue  Canal.  The  hydraulic  pumping  systems  installed 
by  the  Corps  at  the  Orleans  Avenue  Canal  were  sufficient  to  maintain  the 
safe  water  levels  during  a  10-year  rainfall  event.  However,  the  hydraulic 
pumping  systems  installed  at  the  17th  Street  and  London  Avenue  drainage 
canals  could  provide  about  4,000  cfs  and  2,700  cfs,  respectively.  In  order  to 
ensure  that  each  pumping  station  had  the  needed  capacity  to  pump 
enough  water  during  a  10-year  rainfall  event,  the  Corps  used  a  separate 
contract  to  acquire  and  install  an  additional  11  direct  drive  pumps  and  14 
portable  hydraulic  pumps  at  the  17th  Street  Canal,  increasing  the  capacity 
from  about  4,000  cfs  to  about  9,200  cfs.  The  Corps  also  installed  8 
additional  direct  drive  pumps  at  the  London  Avenue  Canal,  increasing  the 
capacity  from  about  2,700  cfs  to  about  5,200  cfs.  Table  1  provides  the  total 
number  of  pumps  and  pumping  capacity  at  the  17th  Street,  London 
Avenue,  and  Orleans  Avenue  Outfall  Canals. 


''a  10-year  rainfall  event  is  a  storm  that  has  a  probability  of  occurring  once  in  10  years,  also 
described  as  having  a  10  percent  chance  of  happening  in  any  year. 
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Table  1 :  Total  Number  of  Pumps  and  Pumping  Capacity  at  the  17th  Street,  London  Avenue,  and  Orleans  Avenue  Outfall 
Canals,  as  of  November  2007 


17th  Street  Canal 


London  Avenue  Canal 


Orleans  Avenue  Canal 


Temporary 


Sewerage 
Capacity  and  Water 
needed  for  Board 
10-year  maximum 


Temporary 


Sewerage  Sewerage 

Capacity  and  Water  Capacity  and  Water 

needed  for  Board  needed  for  Board 

10-year  maximum  Temporary  10-year  maximum 


outfall 

pumps" 

rainfall 

event 

pumping 

capacity 

outfall 

pumps" 

rainfall 

event 

pumping 

capacity 

outfall 

pumps" 

rainfall  pumping 

event  capacity 

Type  of 
pump 

No.  of 
pumps 

Cubic  feet  per  second  (cfs)’’ 

No.  of 
pumps 

Cubic  feet  per  second  (cfs)‘’ 

No.  of 
pumps 

Cubic  feet  per  second  (cfs)” 

Hydraulic 

18 

4,000 

12 

2,700 

10 

2,200 

Direct 

drive 

11 

4,000 

8 

2,500 

N/A 

N/A 

Portable 

hydraulic 

14 

1,400 

N/A 

N/A 

N/A 

N/A 

Total 

43 

9,200" 

7,700 

10,500 

20 

5,200 

5,000 

8,000 

10 

2,200  1,900  2,700 

Source:  GAO  analysis  of  Corps  data. 


“Pump  capacity  numbers  are  approximate  and  are  based  on  rated  pump  vaiues. 

‘’Cubic  feetpersecond  (cfs)  is  a  unitof  measure  for  flow.  One  cfs  is  equivaientto  about449  gaiions 
per  minute  or  646,000  gaiions  per  day. 


The  upper  iimit  on  the  totai  pumping  capacity  at  the  17th  Street  Canai  has  been  set  at  9,200  cfs 
because  of  the  number  of  pumps  atthe  canai. 


Although  these  additional  pumps  allow  the  total  pumping  capacity  at  the 
three  canals  to  maintain  the  capacity  needed  to  pump  water  out  of  the 
canals  during  a  10-year  rainfall  event,  the  capacity  is  still  not  sufficient  to 
match  the  maximum  pumping  capacity  of  the  Sewerage  and  Water  Board’s 
pumps.  As  a  result,  during  a  hurricane  event,  some  flooding  might  occur  in 
some  parts  of  the  city  from  rainfall,  although  it  is  likely  that  this  flooding 
would  be  significantly  less  than  that  which  occurred  from  the  overtopping 
and  breaches  of  the  canal  walls  during  Hurricane  Katrina.  Appendix  11 
provides  the  pumping  capacity  trends  for  the  17th  Street,  London  Avenue, 
and  Orleans  Avenue  drainage  canals  from  June  1,  2006,  through  November 
30,  2007. 
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Commitment  to  Meet 
Schedule  Drove 
Specifications  but 
Resulted  in 
Deficiencies  in  Some 
Key  Contract  Terms 


The  Corps’  efforts  to  develop  the  specifications  for  the  pumping  systems 
were  driven  by  its  commitment  to  have  as  much  pumping  capacity  as 
possible  in  place  at  the  drainage  canals  by  June  1,  2006 — the  start  of  the 
first  Atlantic  hurricane  season  after  Hurricane  Katrina.  Due  to  the 
compressed  schedule  and  the  limited  space  available  for  installation,  and 
based  on  the  limited  market  research  conducted  by  the  Corps’  consultants, 
the  Corps  decided  to  use  60-inch  hydraulic  pumping  systems  rather  than 
alternatives  that  would  have  involved  longer  delivery  schedules  or 
required  more  space.  The  Corps’  consultants  drafted  contract 
specifications  that  closely  matched  those  of  one  supplier,  which,  along 
with  the  60-inch  pumping  system  requirement,  resulted  in  that  supplier 
being  in  the  strongest  position  to  compete  for  the  contract.  Further,  the 
contract  itself  was  not  written  as  precisely  as  it  should  have  been. 
Specifically,  the  original  factory  test  requirements  were  ambiguous,  there 
were  limited  provisions  for  on-site  testing,  and  there  were  no  criteria  for 
acceptance  of  the  pumping  systems  by  the  government. 


Space  and  Schedule 
Requirements  Were  Key 
Considerations  in 
Developing  Specifications 


'The  decisions  made  by  the  Corps  during  the  procurement  of  pumping 
systems  at  three  New  Orleans  drainage  canals  were  driven  largely  by 
space  and  schedule  considerations.  The  Corps  began  the  acquisition 
process  by  contracting  with  two  architectural  and  engineering  consultant 
firms  (consultants)  to  determine  available  technical  options  that  could 
meet  the  Corps’  schedule,  space,  and  pumping  capacity  needs;  conduct  the 
associated  market  research;  and  survey  pump  equipment  suppliers.  On  the 
basis  of  their  technical  analysis,  the  consultants  concluded  that  the  use  of 
hydraulic-driven  pumps  was  the  best  alternative  for  the  Corps  because 
electric-driven  direct  drive  pumps  would  need  auxiliary  equipment  that 
would  require  more  space  for  installation  and  would  have  a  longer  delivery 
time.  They  also  determined  that  using  hydraulic  pumps  less  than  60  inches 
in  diameter  would  require  more  pumps  to  be  installed  and  require  added 
space  to  provide  the  same  amount  of  pumping  capacity.*^ 


^The  60  inches  refers  to  the  diameter  of  the  pumps’  impeller,  which  moves  the  water. 
Simplistically,  pumping  capacity  is  proportional  to  the  square  of  the  pump  diameter.  For 
example,  using  42-inch  hydraulic  pumps  would  require  nearly  twice  the  number  of  pumping 
systems  to  achieve  the  same  capacity  as  60-inch  hydraulic  pumps. 
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Specifications  Closely 
Matched  Those  of  One 
Supplier,  but  Key  Contract 
Provisions  Were  Deficient 


Corps  consultants  drafted  contract  specifications  that  closely  matched 
those  of  one  supplier.  The  consultants  conducted  limited  market  research 
and  found  that  at  least  two  suppliers  had  specifications  for  a  60-inch 
hydraulic  pump.  One  of  those  suppliers  was  MWI,  a  company  that  the 
consultants  had  spoken  with  as  they  were  developing  the  design  for  the 
gates  and  pump  stations  along  the  drainage  canals.  The  consultants  met 
with  MWI  and  also  contacted  at  least  two  other  pump  manufacturers 
regarding  their  pumps.  Of  these  suppliers,  the  consultants  identified  MWI 
as  the  only  supplier  who  had  actually  manufactured  a  60-inch  hydraulic 
pump  with  a  60-inch  impeller,  the  mechanism  that  drives  water  through 
the  system.  Another  pump  manufacturer  had  a  design  for  a  60-inch  pump, 
but  it  included  only  a  54-inch  impeller.  The  consultants  believed  that  MWI 
could  deliver  the  34  60-inch  pumping  systems  that  the  Corps  needed  on 
schedule. 

The  Corps  did  not  have  an  existing  technical  specification  for  a  60-inch 
hydraulic  pump.  The  consultants  drafted  a  specification  for  the  Request 
for  Proposals  (RFP)  based  on  technical  specifications  and  descriptions  of 
the  pumps  contained  in  catalogs  published  by  MWI  and  another 
manufacturer.  The  consultants  told  us  that  they  had  provided  the  Corps 
with  a  generic  specification  because  any  reference  to  a  specific  supplier 
had  been  removed.  However,  our  analysis  of  the  RFP’s  equipment 
specifications  indicates  that  they  more  closely  matched  MWI’s  than  the 
other  manufacturer’s  catalog  descriptions.  In  fact,  the  testing 
specifications  used  for  the  RFP  were  nearly  identical  to  those  published  by 
MWI,  which  included  an  open  sump  test  requirement.  After  the  other 
manufacturer  complained  that  the  open  sump  test  requirement  was 
restrictive  because  only  MWI  had  an  open  sump,  the  Corps  amended  the 
RFP  to  delete  this  requirement.  This  open  sump  test  requirement  was 
incorporated  into  the  contract  at  the  time  of  award,  however,  because  it 
was  offered  by  MWI  as  part  of  its  proposal. 

Other  contractual  testing  and  acceptance  criteria  were  ambiguous, 
inadequate,  or  missing  altogether.  Specifically,  the  contract  did  not  clearly 
state  whether  factory  flow  and  head  testing  was  required  of  each  pump, 
the  on-site  testing  requirement  merely  stated  that  there  should  be  no  leaks, 
and  there  were  no  final  acceptance  criteria  in  the  contract.  Terms  and 
conditions  in  contracts  should  be  clear  and  complete  so  that  the  parties 
fully  understand  their  obligations  and  that  potential  disputes  can  be 
avoided.  To  date.  Corps  and  MWI  have  been  able  to  address  identified 
deficiencies  in  the  contract,  which  were  largely  caused  by  the  perceived 
need  to  move  forward  expeditiously.  However,  the  extent  to  which  these 
or  other  contract  issues  may  lead  to  disputes  between  the  parties  will  not 
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be  known  until  the  time  of  contract  closeout,  currently  scheduled  for  early 
2008. 


The  Corps  Used  a 
Streamlined 
Solicitation  Process 
and  Awarded  the 
Pumping  Systems 
Contract  to  the 
Highest-Rated 
Competitor 


Given  the  need  to  procure  and  install  the  temporary  pumping  systems 
before  the  June  1  start  of  the  2006  hurricane  season,  the  Corps  decided  to 
use  a  streamlined  process  to  contract  for  the  pumping  systems.  Like  most 
other  federal  agencies,  the  Corps  has  statutory  authority  to  use  other  than 
full  and  open  competition  procedures  when  the  agency’s  needs  are  of  an 
unusual  and  compelling  urgency.  Using  this  authority,  the  Corps 
streamlined  parts  of  the  acquisition  process.  The  RFP  was  issued  on 
January  13,  2006,  and  required  that  the  contractors’  proposals  be 
submitted  by  January  18,  just  5  days  later.  Normally  the  solicitation  would 
allow  for  a  response  period  of  at  least  30  days. 

'The  Corps  received  three  proposals  in  response  to  its  RFP.  Suppliers 
submitted  pricing  information  and  technical  proposals  and  made  oral 
presentations  to  the  Corps.**  The  Source  Selection  Evaluation  Board, 
whose  voting  members  consisted  of  three  Corps  officials,  evaluated  offers 
using  four  technical  evaluation  factors  identified  in  the  RFP  in  descending 
order  of  importance:  (1)  technical  approach,  (2)  project  management,  (3) 
past  performance,  and  (4)  small  business  or  small  disadvantaged  business 
participation.  The  solicitation  also  provided  that,  when  combined,  these 
technical  evaluation  factors  were  weighted  approximately  equal  to  price. 

The  Source  Selection  Evaluation  Board  rated  M’WI’s  proposal  significantly 
higher  than  the  other  two  proposals.  MWI’s  proposal  included 
commitments  from  suppliers  and  subcontractors  to  deliver  the  pump 
components  needed  by  MWI  to  assemble  the  pumps.  The  Corps  believed 
MWI  represented  the  best  chance  of  meeting  the  Corps’  critical  deadline  of 
June  1,  2006.  M’iVI  offered  a  price  of  $26.6  million,  which  was  within  2.8 
percent  of  the  government  estimate  of  $25.6  million.  The  contracting 
officer  determined  that  M’WI’s  price  was  fair  and  reasonable  and  awarded 
a  firm,  fixed-price  contract  to  MWI  on  January  27,  2006.  The  contract  also 
contained  an  incentive  of  up  to  $5  million  that  MWI  could  earn  for  early 
delivery.  To  date,  the  Corps  has  increased  the  contract  price  by  about  $6 
million  for  required  pump  modifications  and  for  six  additional  pumping 


^Corps  officials  informed  us  that  videotapes  were  made  of  the  oral  presentations,  but  due 
to  technical  difficulties  the  sound  did  not  record. 
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systems,  bringing  the  total  number  of  hydraulic  pumping  systems  acquired 
to  40.  Figure  4  shows  a  diagram  of  the  hydraulic  pumping  system. 


Figure  4:  Hydraulic  Pumping  System 
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The  Corps  and  the 
Contractors  Have 
Addressed  Pumping 
System  Testing  and 
Performance  Issues 


The  Corps  and  its  contractors  have  addressed  and  corrected  the  pumping 
system  testing  and  performance  issues  identified  by  both  our  May  2007 
report  and  the  ITR.  Factory  testing,  which  occurred  from  March  2006 
through  May  2006,  revealed  several  issues  with  some  components  of  the 
pumping  systems,  and  concerns  were  raised  that  the  pumping  systems 
would  not  perform  as  intended.  On  May  2,  2006,  the  Corps  modified  the 
original  contract,  replacing  the  original  testing  requirements  with  new 
procedures  because  of  schedule  and  performance  concerns.  Beginning  in 
June  2006,  however,  even  though  all  of  the  problems  identified  during 
factory  testing  had  not  been  resolved,  the  systems  were  installed  as 
planned  because  the  Corps  believed  it  was  better  to  have  some  pumping 
capacity  along  the  drainage  canals  during  the  2006  hurricane  season  rather 
than  none.  The  Corps  also  thought  that  most  of  the  issues  identified  during 
factory  testing  could  be  resolved  after  installation.  The  Corps  and  the 
contractors  took  several  steps  to  correct  the  known  performance  issues 
after  installation  and,  as  of  September  2007,  all  of  the  pumping  systems 
have  been  reinstalled  and  all  of  the  outstanding  repairs  have  been 
completed.  According  to  Corps  officials,  the  results  of  on-site  testing  now 
show  that  the  40  hydraulic  pumping  systems  are  fully  operational  and  final 
acceptance  of  the  pumping  systems  is  scheduled  for  the  beginning  of 
calendar  year  2008. 


Factory  Testing 
Requirements  Were 
Modified  to  Focus  on 
Deficiencies  Identified 


On  May  2,  2006,  the  Corps  issued  modification  No.  4,  “revised  test 
procedures,”  to  the  contract.  According  to  the  contracting  officer 
responsible  for  oversight  of  the  pumping  system  contract,  these  revised 
testing  procedures  replaced  the  original  factory  testing  requirements  with 
new  testing  requirements.  The  contract  initially  required  each  pump  and 
hydraulic  power  transmission  system  to  be  factory  pressure  tested 
statically  and  dynamically.^  In  addition,  full-size  flow  and  head  testing  was 
to  be  witnessed  by  the  government  prior  to  shipment  of  the  pumping 
systems.®  The  pump  flow  and  head  testing  was  to  be  conducted  in  an  open 
sump  at  the  manufacturer’s  testing  facility  in  accordance  with  Hydraulic 


According  to  Corps  officials,  static  testing  involves  checking  for  external  leaks  around  the 
seals  of  the  pump  units.  Dynamic  testing  involves  setting  the  pumps  in  water  and  spinning 
the  impeller,  while  using  a  dynamometer  to  determine  the  load  of  the  impeller. 

^According  to  Corps  officials,  flow  and  head  testing  are  conducted  to  determine  the 
predicted  capacity  of  water  a  pump  can  discharge  over  time.  Flow  measures  how  much 
water  can  be  pumped.  Head  measures  how  high  the  water  can  be  pumped. 
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Institute  (HI)  standards'*  and  in  the  presence  of  a  registered  professional 
engineer.  According  to  the  contracting  officer,  modification  No.  4  replaced 
these  testing  procedures.  The  modification  required,  among  other  things, 
testing  the  hydraulic  drive  units  for  a  minimum  of  3  hours  and  utilizing 
previous  model  tests  of  the  pump  design  to  predict  the  pump  capacity. 
Further,  the  modification  required  all  pumps  to  be  pressure  tested  for  90 
minutes. 

According  to  New  Orleans  District  Corps  officials  responsible  for 
oversight  of  the  contract,  the  original  testing  requirements  were 
interpreted  by  a  Corps  inspector  and  the  ITR  to  include  full-size  flow  and 
head  testing  of  each  of  the  pumps.  Furthermore,  the  ITR  concluded  that 
modification  No.  4  did  not  specifically  delete  the  original  testing 
requirements  and,  therefore,  assumed  the  testing  that  had  been  conducted 
did  not  meet  the  contract  requirements  regarding  full-size  flow  and  head 
testing  for  each  pump  in  accordance  with  HI  standards.  Based  on  this 
assumption,  the  ITR  concluded  that  the  contractor  owed  the  government  a 
refund  because  it  had  not  completed  the  testing  required  in  the  contract. 
The  ITR’s  reading  of  the  modification  may  have  overlooked  the 
modification’s  purpose,  however,  which  was  to  adjust  the  required  testing 
to  focus  on  those  elements  of  the  pumps  in  need  of  further  refinement, 
given  the  limited  time  available.  We  believe  the  ITR  may  have  reached  this 
conclusion  because  it  did  not  discuss  the  intent  of  the  “revised  test 
procedures”  modification  to  the  original  testing  requirements  with  either 
the  contracting  officer  or  the  Corps’  technical  officials.  Officials  from  the 
New  Orleans  District  told  us  that  it  was  never  the  intention  of  the  Corps  to 
interpret  the  testing  requirements  as  requiring  every  pump  to  be  full-size 
flow  and  head  tested  in  accordance  with  HI  standards-only  static  and 
dynamic  tests  were  originally  required  of  each  pump.  Corps  technical  and 
contracting  officials  said  the  revised  testing  procedures  contained  in  the 
modification  were  developed  to  focus  on  the  mechanical  issues  that  had 


^HI  is  the  largest  association  of  pump  industry  manufacturers  in  North  America.  The 
institute  provides  industry  standards  for  the  effective  application,  testing,  installation, 
operation,  and  maintenance  of  pumps  and  pumping  systems. 
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been  identified  and,  therefore,  replaced  the  original  testing  requirements, 
which  included  allowing  the  use  of  model  test  results  in  lieu  of  HI  tests.  “ 


Pumping  System  The  Corps  and  the  contractors  have  addressed  and  corrected  the  concerns 

Performance  Issues  Have  raised  about  some  components  of  the  pumping  systems  during  factory  and 

Been  Addressed  by  the  on-site  testing.  As  noted  in  our  May  2007  report  and  in  the  ITR,  the  primary 

Corns  and  the  Contractors  concerns  identified  during  testing  included  undersized  gear  oil  circulation 
^  motors,  hydraulic  motor  vibrations,  the  design  of  the  hydraulic  intake  line, 

suspect  pipe  welds,  and  lower  than  expected  pumping  capacity.  As  a  result 
of  the  concerns  identified  during  testing,  the  Corps  had  no  assurance  that 
the  pumping  systems  would  operate  to  capacity  if  needed  during  the  2006 
hurricane  season.  Nevertheless,  the  pumping  systems  were  installed  as 
planned  because  the  Corps  believed  it  was  better  to  have  some  pumping 
capacity  along  the  drainage  canals  during  the  2006  hurricane  season  rather 
than  none.  On  June  1,  2006,  the  Corps  had  installed  11  pumping  systems, 
and  by  July  2006,  it  had  installed  34,  although  it  is  uncertain  how  much  of 
the  theoretical  capacity  of  these  pumping  systems  would  have  been 
available,  and  for  how  long,  if  needed  during  the  2006  hurricane  season. 
The  Corps  also  believed  that  many  of  the  issues  identified  during  factory 
testing  could  be  resolved  after  installation.  After  installation,  the  Corps 
and  its  contractors  took  several  steps  to  correct  known  performance 
issues  with  the  pumping  systems.  The  main  performance  issues,  and  the 
ways  the  Corps  and  the  contractors  addressed  each  of  them,  are  described 
in  more  detail  below. 

Undersized  Motors 

During  factory  testing,  the  Corps  observed  that  the  gear  oil  circulation 
pump  motors  were  overheating,  which  resulted  in  the  failure  of  some  of 
the  motors.  MWl  determined  that  the  pump  motors  were  too  small.  All  of 
the  motors  were  eventually  replaced  with  larger  gear  oil  circulation 
motors,  resolving  the  problem. 


^°The  Corps  later  became  concerned  that  it  might  not  be  able  to  determine  whether  the 
pumping  system  met  specifications  because  none  of  the  pumping  systems  had  been 
operated  at  design  conditions.  The  Corps,  therefore,  issued  modification  No.  17,  which 
required  the  contractor  to  revert  to  full  performance  and  mechanical  testing  as  required 
under  the  original  contract  for  one  pumping  system.  The  rest  of  the  pumping  systems 
would  be  tested  only  for  mechanical  integrity. 
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Hydraulic  Motor  Vibrations 


During  on-site  testing  in  August  2006,  the  hydraulic  motors  were 
experiencing  greater  than  normal  vibrations.  According  to  the  ITR,  this 
condition  could  have  led  to  the  failure  of  the  equipment.  Initial  analysis  of 
the  problem  indicated  that  there  may  have  been  a  hydraulic  short  in  the 
Rineer  hydraulic  motor  that  drives  the  main  pump  impellers.  The  motor 
manufacturer  made  modifications  to  the  motor,  and  preliminary  testing  of 
the  motors  in  late  August  2006  appeared  to  confirm  that  these 
modifications  eliminated  the  vibrations.  However,  upon  further  testing, 
vibrations  were  still  present  to  varying  degrees.  Additional  on-site  testing 
was  performed,  and  in  late  November  2006,  it  was  determined  that  the 
vibrations  were  due  to  undersized  springs  in  the  Rineer  hydraulic  motors. 
The  motor  manufacturer  replaced  the  undersized  springs  with  heavier 
springs.  According  to  Corps  officials,  on-site  tests  witnessed  by  the 
government  after  the  installation  of  the  new  springs  and  measurements 
conducted  by  a  third-party  contractor  document  that  the  pumping  systems 
now  operate  with  no  apparent  vibration  issues. 

Hydraulic  Intake  Line 

Because  of  concerns  that  the  hydraulic  intake  lines  could  adversely  affect 
pumping  performance,  the  Corps  requested  that  MWI  redesign  and 
reinstall  the  hydraulic  intake  lines  on  all  of  the  pumping  systems.  During 
factory  testing,  the  Corps  observed  a  high  rate  of  failure  of  the  Denison 
hydraulic  pumps  on  the  drive  units.  The  Denison  motors  pump  the 
hydraulic  fluid  from  a  reservoir  to  the  Rineer  motor  which  then  turns  the 
pump  impeller.  A  preliminary  assessment  revealed  that  the  majority  of  the 
issues  identified  in  the  factory  were  caused  by  air  entrainment  (or  dry  run 
condition)  in  the  hydraulic  pumps.  The  dry  run  condition  was  attributed  to 
air  getting  into  the  hydraulic  system  upon  initial  start-up  of  the  drive  unit. 
To  eliminate  the  dry  run  issue,  two  interim  changes  were  made  to  the 
system  until  a  more  permanent  fix  could  be  implemented:  (1)  a  check 
valve  was  installed  on  all  of  the  hydraulic  intake  lines,  and  (2)  the  pump 
start-up  procedure  was  modified  so  that  the  system  was  started  at  a  lower 
speed  and  gradually  increased  to  the  normal  operating  speed. 

The  ITR  concluded  that  the  pumping  systems  would  probably  not  have 
performed  as  designed  because  the  inclusion  of  a  check  valve  would 
require  priming  the  pump  prior  to  start-up  and  the  original  intent  of  the 
design  was  to  allow  for  unmanned  operation  of  the  equipment.  Both  Corps 
and  MWI  officials  stated  that  the  ITR  was  incorrect  in  assuming  that  the 
pumps  would  have  to  be  primed  using  the  check  valve  at  every  start-up. 
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Instead,  these  officials  stated  that  the  pumping  systems  would  have 
operated  as  intended  because  using  this  valve  to  prime  the  Denison  pumps 
is  only  necessary  immediately  after  maintenance  is  performed  on  the 
system.  Additionally,  according  to  a  Lake  Borgne  Levee  District  official, 
this  pump  design  has  been  successfully  used  for  about  20  years  without 
having  to  prime  the  pumps  prior  to  start-up.  Nevertheless,  in  order  to 
ensure  that  air  would  not  be  pulled  into  the  hydraulic  pumps,  causing 
failure  of  the  system,  the  Corps  requested  that  MWI  redesign  the  hydraulic 
intake  system  to  provide  for  a  flooded  suction  without  a  siphon.  Figure  5 
shows  a  drawing  of  the  original  hydraulic  pump  design  with  siphon. 


Figure  5:  Original  Hydraulic  Pump  Design  with  Siphon 


Source:  Moving  Water  Industries  Corporation. 


Corps  officials  from  the  New  Orleans  District  emphasized  to  us  that  the 
redesign  was  requested  to  more  adequately  meet  their  needs,  not  because 
of  concerns  about  the  pumping  systems  operating  as  intended.  MWI 
subsequently  agreed  to  modify  the  design  of  the  hydraulic  intake  line  at 
the  request  of  the  Corps.  According  to  Corps  officials,  by  the  end  of  July 
2007  and  at  its  own  expense,  MWI  had  redesigned  and  reinstalled  the  new 
flooded  suction  design  on  all  40  pumping  systems  (see  fig.  6). 
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Figure  6:  Reinstalled  Hydraulic  Pump  without  Siphon 
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Source:  GAO. 
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Welding 


Because  of  questionable  welds  identified  on  the  pump  housing/^  the  Corps 
decided  to  replace  certain  welds  to  ensure  they  would  not  fail  during 
pump  operations.  Upon  inspection  of  the  pump  housing,  the  Corps 
determined  that  some  of  the  welds  on  the  pump  housing  may  not  be 
sufficient.  While  MWI  provided  the  Corps  with  a  “fit  for  service  letter”  for 
all  of  the  welds  on  the  pump  housing  and  an  extended  warranty,  the  Corps 
decided  that  it  was  prudent  to  replace  the  welds  on  the  pump  housing 
below  the  base  plate  (the  segment  of  the  pump  that  is  below  the  water 
level)  in  order  to  ensure  that  the  welds  would  not  fail  during  pumping 
operations.  All  of  the  necessary  welds  have  been  corrected,  and  the  Corps 
plans  to  negotiate  this  additional  cost  during  contract  closeout. 

Additionally,  issues  were  raised  about  the  adequacy  of  the  welds  on  the 
hydraulic  piping,  which  carries  high  pressure  hydraulic  fluid  from  the 
Denison  pump  to  the  Rineer  motor.  The  hydraulic  piping  was  subsequently 
visually  inspected  and  pressure  tested  to  1.5  times  its  operating  pressure 
as  part  of  the  quality  control  process.  The  testing  results  indicated  that  the 
piping  was  adequate  for  transmitting  power  from  the  diesel  engine  to  the 
water  pump. 

Pumping  Capacity 

Initial  pumping  capacity  testing  indicated  that  the  pumping  systems  may 
not  have  been  performing  at  the  design  capacity  level.  In  April  2006,  MWI 
conducted  full-size  factory  flow  and  head  tests  on  the  hydraulic  pumps.  A 
representative  from  the  Corps’  Engineering  Research  and  Development 
Center  (ERDC)  reviewed  these  test  results  and  concluded  that  the  test 
results  showed  that  the  pumps  would  operate  at  about  96  percent  of  the 
specified  capacity.  However,  according  to  the  ITR,  these  tests  were  not 
conducted  in  accordance  with  HI  standards  and,  therefore,  were  invalid. 

In  August  2006,  on-site  flow  and  head  testing  was  conducted  at  the  canals. 
In  order  to  test  the  pumping  systems,  the  interim  gates  were  closed  and 
water  was  pumped  into  the  canal  by  the  city’s  Sewerage  and  Water  Board 
to  raise  the  water  level  in  the  canal  to  the  elevation  necessary  for  the 
pumping  systems  to  be  tested.  However,  because  adequate  water  levels  in 


^’The  pump  housing  consists  of  60-inch  piping,  which  carries  the  water  pumped  out  of  the 
canal  and  discharges  it  into  Lake  Ponchartrain. 
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the  canal  could  not  be  achieved  to  replicate  design  conditions,  the  pumps 
could  not  reach  a  fully  primed  condition.  The  Corps  decided  to  invert  the 
discharge  pipes  in  order  to  enable  the  pumps  to  reach  a  primed  condition 
with  less  than  design  water  conditions  in  the  lake.  This  facilitated  testing 
of  the  pumping  systems  and  allowed  measurements  to  be  recorded  and 
analyzed. 

In  September  2006,  a  representative  from  ERDC  was  consulted  and 
performed  on-site  flow  and  head  tests  of  pumping  systems  at  the  London 
Avenue  canal.  A  month  later  the  Corps  and  ERDC  performed  the  same 
tests  at  the  17th  Street  canal.  Data  collected  from  these  on-site  tests 
revealed  that  the  pumping  systems  were  working  near  the  appropriate 
capacity.  Based  upon  the  on-site  testing  results  and  upon  suggestion  from 
the  ERDC  representative,  all  of  the  discharge  pipes  at  all  of  the  canals 
were  inverted  and  cut  at  a  30  degree  angle,  which  allows  the  pumps  to 
prime  at  lower  canal  water  elevations  and  enhances  the  flow  rates  (see  fig. 

7). 
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Figure  7:  Discharge  Pipes  at  the  London  Avenue  Canal 


Source:  GAO. 


In  November  2006,  another  full-size  factory  flow  and  head  test  was 
conducted  by  MWI  and  ERDC.  However,  due  to  constraints  at  the  testing 
facility,  the  full-size  factory  test,  which  was  done  in  consultation  with  the 
ERDC  representative,  was  completed  with  deviations  from  the  HI 
standards.  This  test  revealed  that  the  pumping  capacity  ranged  from  93.6 
to  97.6  percent  of  the  design  specification  and  performed  without 
problems  during  the  2  days  of  testing.  According  to  Corps  officials,  MWI 
further  agreed  to  construct  a  model  test  to  confirm  the  pumping  systems 
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would  perform  within  HI  standards.  In  September  2007,  a  Corps  ERDC 
official  witnessed  a  model  test  conducted  by  MWI  and  prepared  a  report, 
which  concluded  that  the  pumping  systems  would  operate  at  98.6  percent 
of  the  design  capacity.  According  to  Corps  officials,  these  results  are 
within  acceptable  limits  and  any  issues  remaining  with  the  final  pumping 
capacity  will  be  negotiated  at  contract  closeout. 


Pumping  System  According  to  Corps  officials,  the  Corps  plans  to  make  final  acceptance  of 

Near  the  pumping  systems  during  the  beginning  of  calendar  year  2008.  The 

original  pumping  system  contract  lacked  clearly  defined  on-site  testing 
procedures,  requiring  only  that  the  pumps  and  hydraulic  equipment  be 
tested  for  leaks.  In  light  of  the  various  issues  surrounding  the  pumping 
systems,  the  Corps  and  MWI  agreed  that  it  was  necessary  to  show  that  all 
of  the  pumping  systems  could  operate  at  a  steady  state  after  installation. 
According  to  Corps  and  MWI  officials,  a  major  challenge  with  on-site 
testing  of  the  pumping  systems  is  simulating  the  amount  of  water  that 
would  be  present  in  the  canals  and  the  lake  during  a  storm  event.  Under 
normal  conditions,  when  there  are  low  water  levels  in  the  canals,  it  is  not 
possible  to  test  each  pump  system  for  an  extended  period  of  time,  and  any 
tests  conducted  cannot  approach  the  design  capacity  of  the  pumping 
systems.  Due  to  this  limitation,  the  Corps  subsequently  developed  specific 
pumping  system  acceptance  testing  procedures  that,  among  other  things, 
include  running  each  pumping  system  continuously  for  2  hours.  Corps 
officials  told  us  that  because  most  of  the  issues  associated  with  the 
pumping  systems  occurred  within  the  first  45  minutes  of  operation,  the  2- 
hour  testing  period  for  each  pumping  system  was  sufficient. 

In  its  June  2007  report,  the  ITR  team  concluded  that  at  the  time  of  their 
review  in  September  2006,  the  pumping  systems  would  not  perform  as 
intended  because  of  issues  encountered  in  factory  testing  in  early  2006. 
Since  September  2006,  there  have  been  a  number  of  analyses,  changes, 
and  additional  testing  of  the  pumping  systems  to  address  these  earlier 
concerns.  For  example,  between  November  2006  and  September  2007,  the 
Corps  had  completed  all  of  the  repairs  that  were  outstanding  at  the  end  of 
the  2006  hurricane  season  and  which  were  noted  in  the  ITR,  and 
reinstalled  all  40  pumping  systems.  In  addition,  as  of  September  2007,  each 
pumping  system  had  been  successfully  tested  on  site  for  at  least  2  hours, 
providing  greater  assurance  that  they  will  perform  as  designed  during 
future  hurricane  seasons.  On  September  27,  2007,  GAO  officials  witnessed 
the  pumping  systems  performing  at  both  the  17th  Street  and  London 
Avenue  Canals  (see  fig.  8).  According  to  Corps  officials,  because  all  of  the 
outstanding  repairs  have  been  completed  and  on-site  testing  indicates  that 


Acceptance  Is 
Completion 
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the  system  is  now  fully  operational,  final  acceptance  of  the  pumping 
systems  and  the  contract  closeout  is  expected  to  be  completed  early  in 
calendar  year  2008. 


Figure  8:  Hydraulic  Pumping  Systems  Performing  at  the  17th  Street  Canal,  September  27,  2007 


Source:  GAO. 
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Contract  Files 
Remained  Incomplete 
for  Months,  but 
Currently  Contain 
Required 

Documentation  for 
the  Type  and  Value  of 
Procurement 


Contract  files  for  the  pumping  systems,  although  incomplete  at  the  time  of 
the  ITR  review,  now  contain  the  required  documentation  for  the  type  of 
contract  and  the  value  of  the  associated  modifications.  In  a  number  of 
cases,  however.  Corps  officials  inserted  required  documentation  in  the 
contract  files  several  months  after  modifications  were  issued  and  only 
after  the  ITR  reported  its  findings.  While  the  ITR  correctly  noted  the 
absence  of  some  forms  of  required  documentation,  we  found  that  much  of 
the  documentation  specifically  cited — including  requests  for  proposals, 
independent  government  estimates,  certified  cost  or  pricing  data,  technical 
analyses,  and  price  negotiation  memorandums — was  not  required  for  the 
modifications  in  question.  In  addition,  while  the  ITR  found  that  it  appeared 
as  though  the  contractor  developed  the  scope  of  work  and  pricing  for 
some  of  the  modifications  without  a  subsequent  analysis  by  the  Corps,  we 
found  no  instance  of  this  occurring.  Rather,  our  review  found  that,  for 
most  of  the  contract  modifications  there  was  extensive  back  and  forth 
discussion,  usually  by  e-mail,  between  officials  from  the  Corps  and  MWI. 


Significant  Documentation 
Added  to  the  Contract 
Files  Only  after  the  ITR 
Review 


The  ITR  team  reviewed  18  of  the  first  30  contract  modification  files  and 
reported  that  many  lacked  significant  documentation.  Specifically,  the  ITR 
identified  13  modification  files  with  deficiencies — most  pertaining  to 
documentation  of  the  Corps’  determination  of  fair  and  reasonable  pricing. 
Our  review  confirmed  that  significant  documentation  was  added  to  the 
files  only  after  the  ITR  team  issued  its  report.  We  reviewed  the  files  for  the 
32  post-award  modifications,  focusing  in  depth  on  the  files  related  to  the 
13  modifications  found  by  the  ITR  team  to  contain  deficiencies,  as  well  as 
2  additional  modifications  that  were  issued  after  our  May  2007  report  and 
the  ITR  review.  Of  the  modifications  we  reviewed  in  depth,  10  contained 
internal  memorandums,  prepared  by  the  contracting  officer  after  the  fact, 
to  document  price  reasonableness  or  the  events  supporting  the 
modification.  Another  2  modifications  contained  undated  memorandums 
of  price  reasonableness  signed  by  the  contracting  officer.  Finally,  of  the 
eight  purchase  request  and  commitment  forms  on  file,  five  were  prepared 
on  the  same  date  to  retroactively  document  the  availability  of  funds  for 
modifications  that  were  issued  9  to  17  months  earlier.  Documentation  in 
some  of  the  files,  however,  suggests  that  the  availability  of  funds  was 
determined  through  other  means  at  the  time  the  modifications  were 
signed. 


In  response  to  the  ITR,  the  Corps’  contracting  officer  acknowledged  that 
the  contract  files  could  have  been  better  managed  but  stated  the  Corps  felt 
it  was  more  important  to  get  the  pumps  installed  in  a  timely  manner.  In 
order  to  do  this,  the  Corps  issued  the  modifications  with  the  intention  of 
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settling  all  outstanding  issues  with  the  contractor  before  closing  out  the 
contract.  The  Corps  agreed  with  the  ITR,  however,  that  certain 
documentation  was  missing  and  took  corrective  actions  to  complete  the 
files.  The  contracting  officer,  whom  the  ITR  team  did  not  meet  with  for 
their  review,  noted  that  because  many  of  the  people  working  on  the 
pumping  systems  procurement  were  rotating  through  the  District  Office, 
they  may  not  have  completed  or  submitted  all  of  the  necessary  paperwork 
before  leaving.  Even  though  it  is  currently  complete,  preparing 
documentation  months  after  an  event  occurs  increases  the  likelihood  that 
the  documentation  may  contain  inaccuracies  or  ambiguities,  which  make 
it  difficult  to  resolve  any  disputes  that  may  arise. 


Many  Specific  Documents  As  of  October  2007,  the  contract  modification  files  appeared  up  to  date 

Cited  by  the  ITR  as  Missing  consistent  with  Federal  Acquisition  Regulation  (FAR)  requirements. 

Were  Not  Required  While  the  ITR  correctly  noted  the  absence  of  some  forms  of  required 

documentation,  much  of  the  documentation  specifically  cited  by  the  ITR — 
including  requests  for  proposals,  independent  government  estimates, 
certified  cost  or  pricing  data,  technical  analyses,  and  price  negotiation 
memorandums — was  not  required  for  the  modifications  in  question.^^  In 
some  respects,  it  appears  the  ITR  treated  the  pumping  systems  contract  as 
if  it  were  for  construction  rather  than  supplies.  Different  documentation 
requirements  apply  to  these  types  of  contracts. 

Ten  of  the  modifications  we  reviewed  in-depth  increased  contract  costs 
and,  therefore,  required  documentation  of  fair  and  reasonable  pricing.^ 
While  independent  government  estimates  are  one  technique  that  can  be 
used  to  analyze  price  and  are  required  for  construction  contracts,  they  are 
not  specifically  required  for  supply  contracts,  such  as  the  contract  for  the 
pumping  systems.  Nonetheless,  the  Corps  obtained — and  included  in  the 
files  after  the  ITR  review — independent  government  estimates  for  six  of 
the  modifications. 

None  of  the  10  modifications  with  additional  costs  that  we  reviewed  in- 
depth  required  the  contractor  to  provide  certified  cost  or  pricing  data. 
Specifically,  we  found  that  7  of  the  modifications  fell  under  the  threshold 


^^The  documentation  cited  by  the  ITR  would  not  have  impacted  the  weaknesses  we 
identified  earlier  in  this  report. 

^^Fourteen  of  the  32  modifications  resulted  in  increased  contracting  costs. 
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requiring  cost  or  pricing  data.“  The  contracting  officer  determined  that 
cost  or  pricing  data  was  not  required  for  another  modification  because  it 
combined  separately  priced  changes  from  2  previous  modifications  that 
were  each  below  the  threshold.  Finally,  for  2  modifications  related  to  the 
purchase  of  six  additional  pumps,  the  contracting  officer  concluded  that 
adequate  price  competition  existed  from  the  base  contract  and,  therefore, 
additional  pricing  data  was  not  required.  At  least  some  information  on 
pricing  provided  by  the  contractor  was  included  in  the  files  for  all  10  of  the 
modifications  that  involved  additional  costs. 

According  to  the  FAR,  when  contractor  certified  cost  and  pricing  data  are 
not  required,  price  analysis  shall  be  used  to  determine  a  fair  and 
reasonable  price.  While  the  FAR  provides  numerous  analysis  techniques, 
including  the  use  of  independent  government  estimates,  it  does  not  require 
the  use  of  any  one  method.  For  8  of  the  modifications  we  reviewed,  the 
Corps’  contracting  officer  documented  price  analysis  and  negotiations 
with  the  contractor  through  signed  internal  memorandums  for  the  files, 
and  for  2  modifications,  used  price  negotiation  memorandums.'"^  In 
addition,  while  not  required,  the  Corps  obtained  internal  technical 
analyses  for  3  of  the  modifications  we  reviewed  in  depth  to  determine  the 
reasonableness  of  MWFs  proposals.  Table  2  summarizes  GAO’s  analysis  of 
the  ITR’s  findings  regarding  missing  documentation  in  the  contract  files. 


"The  threshold  for  certified  cost  and  pricing  data  was  increased  from  $550,000  to  $650,000 
on  September  28,  2006. 

'^While  price  negotiation  memorandums  can  be  used  to  document  price  analysis  findings, 
such  as  fair  and  reasonable  pricing,  they  are  not  specifically  required  for  documenting  the 
agreement  negotiated  between  the  contractor  and  agency. 
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Table  2:  GAO  Analysis  of  the  ITR  Findings 

Documentation  cited  as  missing  by  ITR 

GAO  analysis  and  applicable  FAR  references 

Request  for  proposal 

Not  always  required  for  contract  modifications.  Under  the  FAR,  the  Standard  Form  (SF) 

30,  Amendment  of  Sollcltatlon/Modlflcatlon  of  Contract  may  be  used  at  the  discretion  of 
the  contracting  officer.  FAR  section  15.210(b). 

Independent  government  estimate 

A  comparison  of  contractor-proposed  prices  with  an  Independent  government  estimate  Is 
one  of  many  examples  of  price  analysis  techniques,  but  Is  not  specifically  required  for 
contract  modifications.  FAR  section  15.404-l(b). 

Contractor  certified  cost  or  pricing  data 

Certified  cost  or  pricing  data  Is  required  for  modifications  of  any  negotiated  contract 
expected  to  exceed  the  current  threshold  of  $650,000  except  when  adequate  price 
competition  exists  or  when  commercial  Items  are  being  acquired.  In  addition,  this 
requirement  does  not  apply  when  unrelated  and  separately  priced  changes  below  the 
threshold  are  Included  for  administrative  convenience  In  the  same  modification.  FAR 
sections  15.403-1,  15.403-4. 

Price,  cost,  or  technical  analysis 

P  rice  analysis  shall  be  used  when  contractor  cost  or  pricing  data  are  not  required.  Various 
price  and  cost  analysis  techniques  are  listed  In  the  FAR,  Including  the  use  of  Independent 
government  estimates.  Flowever,  none  of  the  listed  techniques  are  specifically  required. 

In  addition,  the  contracting  officer  may,  but  Is  not  specifically  required  to,  request  that 
personnel  with  specialized  knowledge,  skills,  experience,  or  capabilities  perform  technical 
analyses  of  contractor  proposals  to  determine  reasonableness.  FAR  section  15.404-1. 

Price  negotiation  memorandum 

While  price  negotiation  memorandums  can  be  used  to  document  price  analysis  findings, 
such  as  fair  and  reasonable  pricing,  they  are  not  specifically  required  for  documenting  the 
agreement  negotiated  between  the  contractor  and  agency.  These  memorandums 
document  the  principal  elements  of  the  negotiated  agreement.  FAR  section  15.406-3. 

Source:  GAO. 

File  Documentation 
Establishes  Independence 
of  Corps  Decision  Making 

In  addition  to  contract  documentation  issues,  the  ITR  also  reported  that  it 
appeared,  in  some  circumstances,  as  though  the  contractor  developed  the 
scope  of  work  and  pricing  for  the  modifications  without  a  subsequent 
analysis  by  the  Corps.  We  found  no  instance  of  this  occurring.  Rather,  our 
review  of  the  files  indicate  that,  for  most  of  the  contract  modifications, 
there  was  extensive  back  and  forth  discussion,  usually  by  e-mail,  between 
personnel  from  the  Corps  and  MWI.  These  discussions  focused  on  the 
causes  of  and  solutions  to  technical  issues,  as  well  as  the  costs  of 
corrective  actions. 

While  each  of  the  modifications  was  unique,  modification  No.  2  is 
illustrative  of  many  of  the  contract  modifications  we  reviewed. 

Specifically,  shortly  after  award  of  the  contract,  the  Corps  determined  that 
it  needed  the  capability  to  control  the  pumps  from  a  remote  location,  since 
in  the  event  of  a  hurricane  the  operator  would  be  required  to  seek  shelter 
in  a  control  booth.  The  Chief  of  Engineering  from  the  Corps  prepared  a 
request  to  modify  the  contract  to  require  master  pump  control  panels.  The 
request  contained  detailed  specifications  of  what  was  required  and 
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estimated  that  the  additional  cost  would  be  $160,000.  The  contracting 
officer  sent  the  request  to  MWl  and  asked  for  a  cost  proposal.  MWI  replied 
through  an  e-mail  that  repeated  the  specifications  provided  to  it  by  the 
Corps  and  offered  a  price  of  $188,699.  The  Corps  requested  additional 
support  for  the  price,  and  MWl  responded  with  a  copy  of  the  quote  it  had 
received  from  its  supplier,  pricing  for  MWI’s  markup,  and  the  additional 
work  MWI  would  perform.  A  Corps  engineer  reviewed  this  information 
and  informed  the  contracting  office  that  MWFs  proposed  price  was 
reasonable.  The  Corps  issued  a  contract  modification  with  the 
specifications  it  developed  at  the  price  quoted  by  MWI.  As  was  the  case  for 
a  number  of  modifications,  there  was  no  contemporaneous  price 
reasonableness  document  signed  by  the  contracting  officer;  rather,  an 
undated  “after  the  fact”  memorandum  concluding  that  MWI’s  price  for  the 
modification  was  reasonable  was  added  to  the  file. 


The  Corps  Has  Not 
Overpaid  the  Contract 
and  Has  Plans  to 
Reconcile  Payments 
Made  in  Error 


As  of  October  31,  2007,  the  Corps  had  paid  the  contractor  about  $30.5 
million  of  the  $33  million  contract  for  the  40  hydraulic  pumping  systems 
and  has  plans  for  reconciling  mistaken  payments  it  made.  The  Corps  made 
payments  to  the  contractor  only  after  receiving  invoices  from  the 
contractor  for  delivered  items  and  services.  In  most  cases,  the  Corps  only 
paid  80  percent  of  each  invoice  and  held  the  other  20  percent  as  retained 
funds  in  order  to  ensure  the  contractor  was  not  overpaid  and  that  any 
performance  issues  were  addressed. 


The  ITR  identified  a  few  instances  where  the  contractor  had  received 
payment  more  than  once  for  the  same  item.  Our  review  confirmed  that  this 
did  occur.  We  found,  however,  that  these  payments  were  made  in  error  by 
the  Corps  and  did  not  indicate  any  improper  behavior  on  the  contractor’s 
part.  Specifically,  on  December  6,  2006,  the  Corps  received  one  invoice 
requesting  payment  for  three  drive  units  and  three  pumps  valued  at  about 
$2.2  million  because  they  were  complete,  and  MWI  believed  that  they 
could  be  delivered  if  the  Corps  wanted  them  at  that  time.  On  the  same  day, 
the  Corps  notified  MWI  that  it  could  not  pay  for  the  pumps  and  drive  units 
until  they  were  actually  delivered.  MWI  then  e-mailed  the  Corps, 
requesting  that  they  ignore  the  original  invoice  and  stating  that  they  would 
send  new  invoices  for  the  drive  units  and  pumps  upon  shipment.  The 
Corps  subsequently  received  three  separate  invoices,  each  requesting 
payment  for  one  drive  unit  and  one  pump.  However,  the  Corps  paid  all  of 
the  invoices,  including  the  invoice  that  the  contractor  told  them  to  ignore. 
As  a  result,  the  Corps  paid  twice  for  the  same  three  pumps  and  three  drive 
units.  According  to  the  Corps’  contracting  officer,  the  duplicate  payments 
will  be  corrected  by  deducting  the  balance  from  withheld  funds  and  not 
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paying  some  outstanding  invoices.  Our  review  found  no  other  instances 
where  duplicate  payments  were  made  to  the  contractor.  We  also  found  14 
instances  where  the  contractor  sent  invoices  to  the  Corps  for  work 
completed,  which  have  not  been  paid.  The  net  effect  is  that  the  contractor 
has  not  been  overpaid  under  the  contract. 

On  June  8,  2007,  the  Corps  sent  a  letter  to  MWI  providing  an  explanation 
as  to  why  the  Corps  had  not  paid  these  outstanding  invoices,  and 
describing  how  the  Corps  planned  to  reconcile  the  duplicate  payments 
made  in  January  and  February  2007  by  subtracting  the  amoimt  of  the 
outstanding  invoices  from  any  additional  invoices  it  received.  From  July 
through  October  2007,  the  Corps  made  four  additional  payments  to  the 
contractor  from  the  payments  it  had  withheld,  totaling  about  $1.8  million. 
The  Corps  has  still  not  made  final  payment  for  the  outstanding  amount 
remaining  on  the  contract.  In  addition,  the  Corps  has  withheld  payments 
related  to  an  early  delivery  incentive  of  approximately  $5  million  until  the 
final  acceptance  of  the  pumping  systems.  “  According  to  Corps  officials, 
the  final  payment  and  reconciliation  of  the  contract,  including  any 
incentive  payments  or  penalties,  will  be  settled  with  the  contractor  after 
final  acceptance  of  the  pumping  systems.  The  Corps  expects  this  to  take 
place  in  the  early  part  of  calendar  year  2008. 


Conclusion 


The  Corps’  actions  in  awarding  and  administering  the  pumping  system 
contract  were  generally  in  accordance  with  federal  requirements. 

However,  in  its  haste  to  award  the  contract  and  acquire  and  install  the 
pumps,  the  Corps  did  not  develop  a  contract  that  was  clear  and  precise 
with  respect  to  testing  and  acceptance  criteria  and  did  not  always 
promptly  prepare  required  contract  related  documents.  In  some  cases,  this 
has  led  to  uncertainties  about  exactly  what  was  required  of  the  contractor 
to  comply  with  the  contract’s  terms  and  conditions.  This  also  creates  the 
potential  for  contract  disputes,  which  can  be  difficult,  expensive,  and  time- 
consuming  to  resolve.  In  addition,  in  those  cases  where  required 
documents  were  prepared  “after  the  fact,”  there  is  an  increased  likelihood 
that  documents  prepared  months  after  events  have  occurred  may  contain 
inaccuracies  as  memories  have  faded  and  key  personnel  may  have  moved 
on  to  other  positions. 


^®The  incentive  clause  also  has  a  penalty  for  late  delivery  of  $1,700  per  pump,  per  day.  The 
Corps  will  determine  whether  or  not  a  penalty  will  be  assessed  as  part  of  the  close-out 
process. 
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Recommendations  for 
Executive  Action 


While  we  recognize  that  this  procurement  was  conducted  under  exigent 
circumstances,  we  believe  that  the  procedures  used  by  the  Corps  could  be 
improved  for  future  procurements.  For  this  reason  we  recommend  that  the 
Secretary  of  Defense  direct  the  Commanding  General  and  Chief  of 
Engineers  of  the  U.S.  Army  Corps  of  Engineers  to; 


•  take  steps,  through  additional  guidance  or  otherwise,  to  reinforce  the 
importance  of  adherence  to  sound  acquisition  practices,  even  during 
expedited  procurements,  including  ensuring  that  important  contract 
provisions,  such  as  any  required  testing,  are  clear  so  that  the  contractor 
and  the  government  understand  what  conditions  or  criteria  must  be  met 
for  successful  completion  of  the  contract;  and 

•  develop  procedures  to  ensure  that  any  required  contract-related 
documentation,  including  that  related  to  contract  pricing,  is  completed 
and  filed  within  a  reasonable  period  of  time. 


Agency  Comments 
and  Our  Evaluation 


The  Department  of  Defense  provided  written  comments  on  a  draft  of  this 
report,  which  are  reprinted  in  appendix  III.  The  Department  of  Defense 
concurred  with  our  recommendations  and  provided  information  on  what 
actions  it  would  take  to  address  them.  Concerning  our  recommendation 
to  adhere  to  sound  acquisition  practices,  the  Department  of  Defense  said 
the  Secretary  of  Defense  will  direct  the  Corps  to  send  guidance  to  all 
Corps  offices  emphasizing  the  need  for  clearer  technical  specifications  so 
that  the  contractor  and  government  understand  what  conditions  or  criteria 
must  be  met  for  successful  contract  completion.  To  address  our 
recommendation  to  ensure  more  timely  completion  of  required  contract 
file  documentation,  the  Department  of  Defense  said  the  Secretary  of 
Defense  will  direct  the  Corps  to  review  and  revise  as  necessary  current 
policies  and  regulations.  The  Department  of  Defense  also  provided  us 
with  technical  comments,  which  we  have  incorporated  throughout  the 
report,  as  appropriate. 
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As  agreed  with  your  office,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
report  date.  At  that  time,  we  will  send  copies  to  interested  congressional 
committees;  the  Secretary  of  Defense;  and  the  Commanding  General  and 
Chief  of  Engineers  of  the  U.S.  Army  Corps  of  Engineers.  We  will  also  make 
copies  available  to  others  on  request.  In  addition,  the  report  will  be 
available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  or  any  of  your  staff  have  any  questions  about  this  report,  please 
contact  one  of  us  at  (202)  512-3841  or  mittala@gao.gov,  (202)  512-4841  or 
woodsw@gao.gov,  or  (202)  512-6923  or  dornt@gao.gov.  Contact  points  for 
our  Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on 
the  last  page  of  this  report.  Key  contributors  to  this  report  are  listed  in 
appendix  IV. 

Sincerely  yours. 


Anu  K.  Mittal 

Director,  Natural  Resomces  and  Environment 


William  T.  Woods 

Director,  Acquisition  and  Sourcing  Management 


Terrell  G.  Dorn 

Director,  Physical  Infrastructure 
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Appendix  I:  Scope  and  Methodology 


To  evaluate  efforts  by  the  U.S.  Army  Corps  of  Engineers  (Corps)  to  solicit, 
award,  and  administer  the  pumping  system  contract,  we  reviewed  the 
Corps’  plans  for  the  interim  gates  and  temporary  pumping  systems 
consisting  of  the  40  hydraulic  pumps  installed  at  the  three  New  Orleans 
drainage  canals.  We  also  reviewed  applicable  Federal  Acquisition 
Regulation  criteria,  especially  pertaining  to  contract  pricing;  the  contract 
and  specifications;  e-mails,  correspondence,  and  other  supporting 
documentation  related  to  the  solicitation  and  award  of  the  contract; 
factory  and  on-site  test  results;  performance  requirements;  the  32  contract 
modifications  and  supporting  documentation;  the  Mississippi  Valley 
Division  (MVD)  Independent  Team  Report  (ITR);  the  Corps  project 
delivery  team’s  response  to  the  ITR;  contractor  invoices  and  payment 
records;  and  the  Corps’  plans  for  increasing  pumping  capacity  through 
2007. 

We  also  visited  the  17th  Street,  London  Avenue,  and  Orleans  Avenue 
Canals  and  observed  the  operation  of  the  pumping  systems.  We 
interviewed  contracting  and  program  officials  from  (1)  Corps 
Headquarters,  New  Orleans  District,  other  Corps  districts,  and  members  of 
the  MVD’s  technical  review  team  related  to  the  contract  and  pump 
performance;  (2)  Moving  Water  Industries  Corporation  and  two  other 
pump  suppliers  that  bid  on  the  solicitation;  and  (3)  the  architectural  and 
engineering  consulting  firms  under  contract  with  the  Corps  that 
researched  available  pumping  system  alternatives,  including  qualified 
pump  manufactures,  pump  delivery  timelines,  and  costs,  and  that  helped 
design  the  canal  gates  and  pumping  stations.  We  conducted  our  work  from 
September  through  December  2007  in  accordance  with  generally  accepted 
government  auditing  standards. 
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Appendix  II:  Pumping  Capacity  for  the  Three 
Drainage  Canals 


Figure  9:  Pumping  Capacity  Trend  for  the  17th  Street  Canal,  June  1,  2006  through 
November  30,  2007 
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Source:  GAO  analysis  of  U.S.  Army  Corps  of  Engineers' data. 
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Appendix  II:  Pumping  Capacity  for  the  Three 
Drainage  Canais 


Figure  10:  Pumping  Capacity  Trend  for  the  London  Avenue  Canal,  June  1,  2006 
through  November  30,  2007 
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Source:  GAO  analysis  of  U.S.  Army  Corps  of  Engineers' data. 
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Appendix  II:  Pumping  Capacity  for  the  Three 
Drainage  Canais 


Figure  11 :  Pumping  Capacity  Trend  for  the  Orleans  Avenue  Canal,  June  1,  2006 
through  November  30,  2007 
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Source:  GAO  analysis  of  U.S.  Army  Corps  of  Engineers' data. 
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Appendix  III:  Comments  from  the 
Department  of  Defense 


DEPARTMENT  OF  THE  ARMY 

U.S.  ARMY  CORPS  OF  ENGINEERS 
441  G  ST.  NW 

WASHINGTON,  D.C.  20314-1000 


December  18,  2007 


Internal  Review  Office 


Ms.  Ann  K.  Mittal 
Director,  Natural  Resources 
And  Environment 
Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  D.C,  20548 

Dear  Ms.  Mittal: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report  08-288, 
‘ARMY  CORPS  OF  ENGINEERS:  Known  Performance  Issues  with  New  Orleans  Drainage 
Canal  Pumps  Have  Been  Addressed,  but  Guidance  on  Future  Contracts  Is  Needed,’  dated 
December  4,  2007,  (GAO  Code  360879). 

We  appreciate  the  opportunity  for  reviewing  the  draft  report.  Command  comments  to 
the  report  recommendations  are  contained  in  the  attachment. 

If  you  have  any  questions  or  require  additional  information,  please  contact  Ms.  Alicia 
Matias,  202-761-4573,  alicia.s.matias@usace.anny.mil. 


Enclosure 
As  Stated 


DONNA  F.  JOHN§12iN 
Acting  Chief  Audit  Executive 
HQ  Internal  Review  Office 
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Appendix  III:  Comments  from  the  Department 
of  Defense 


GAO  DRAFT  REPORT  08-288,  ‘ARMY  CORPS  OF  ENGINEERS: 
KNOWN  PERFORMANCE  ISSUES  WITH  NEW  ORLEANS 
DRAINAGE  CANAL  PUMPS  HAVE  BEEN  ADDRESSED,  BUT 
GUIDANCE  ON  FUTURE  CONTRACTS  IS  NEEDED,’  DATED 
DECEMBER  4,  2007,  (GAO  CODE  360879) 


DEPARTMENT  OF  DEFENSE  RESPONSE  TO  THE 
RECOMMENDATIONS 


Recommendation:  We  recommend  that  the  Secretary  of  Defense  require  the  Chief  of  Engineers 
to: 

Take  steps,  through  additional  guidance  or  otherwise,  to  reinforce  the  importance 
of  adherence  to  sound  acquisition  practices  even  during  expedited  procurements, 
including  ensuring  that  important  contract  provisions,  such  as  any  required  testing, 
are  clear  and  that  the  contractor  and  the  government  understand  what  conditions  or 
criteria  must  be  met  for  successful  completion  of  the  contract. 

DOD  RESPONSE:  CONCUR:  The  Secretary  of  Defense  will  direct  the  U.S.  Army  Corps  of 
Engineers  to  send  a  memo  to  all  Coips  offices  reinforcing  the  importance  of  adherence  to  sound 
engineering  practices  even  during  expedited  procurements.  The  memo  will  relate  lessons 
learned  and  emphasize  the  need  for  technical  specifications,  such  as  those  required  for  equipment 
testing,  to  be  clear  so  that  the  contractor  and  the  government  understand  what  conditions  or 
criteria  must  be  met  for  successful  completion  of  the  contract.  The  anticipated  date  to  transmit 
the  memo  is  1 1  January.  2008. 

Develop  procedures  to  ensure  that  any  contract  related  documentation,  including  that 
related  to  contract  pricing,  is  completed  and  filed  within  a  reasonable  period  of  time. 

DOD  RESPONSE:  CONCUR:  The  Secretary  of  Defense  will  direct  the  U.S.  Army  Corps  of 
Engineers  to  review  and  revise,  as  necessary,  current  policies  and  regulations  to  ensure  that  a 
reasonable  period  of  time  is  identified  for  completing  and  filing  contract  documents.  Estimated 
completion  date  is  30  May,  2008. 
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Appendix  IV:  GAO  Contacts  and  Staff 
Acknowledgments 


GAO  Contacts 


Anu  K.  Mittal,  (202)  512-3841  or  mittala@gao.gov 
William  T.  Woods,  (202)  512-4841  or  woodsw@gao.gov 
Terrell  G.  Dom,  (202)  512-6923  or  dornt@gao.gov 


Staff 

Acknowledgments 


In  addition  to  the  contacts  named  above,  Ed  Zadjura,  Assistant  Director; 
Matthew  Reinhart;  Katherine  Trimble;  Christine  Frye;  James  Dishmon; 
Rich  Johnson;  Marie  P.  Aheam;  and  Kenneth  E.  Patton  made  significant 
contributions  to  this  report. 


(360879) 
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GAO’S  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  pohcies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  rehability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday,  GAO  posts 
newly  released  reports,  testimony,  and  correspondence  on  its  Web  site.  To 
have  GAO  e-mail  you  a  list  of  newly  posted  products  every  afternoon,  go 
to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each. 

A  check  or  money  order  should  be  made  out  to  the  Superintendent  of 
Documents.  GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or 
more  copies  mailed  to  a  single  address  are  discounted  25  percent.  Orders 
should  be  sent  to: 

U.S.  Government  Accoimtability  Office 

441  G  Street  NW,  Room  LM 

Washington,  DC  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Gloria  Jarmon,  Managing  Director,  jarmong@gao.gov,  (202)  512-4400 

U.S.  Government  Accoimtability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 


Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 
U.S.  Government  Accoimtability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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